1 Elo Ratings for CFB

We can now use historical CFB games to learn Elo Ratings for college football teams, which we will then use to predict upcoming games and seasons. The data we’re using is at the game level, in which we have features indicating the season, the week, the location, the teams, and the score.

We also have additional features on team conferences, attendance, start time, betting line, as well as information from an existing Elo Model. We’ll set that aside for now, as our goal here is to go through the process of developing our own Elo model.

We lose some information the further we go back, but we have outcomes of games back to the very first game between Rutgers and Princeton in 1869. Note: We have more games in recent years due to having more data on FCS teams in these seasons.

We could start this analysis going back all the way to 1869 as the starting point, but I’ll start at 1970 for now since I’m at least somewhat familiar with teams in this era and will be able to eyeball some of the results

I’ll set every team FBS’s initial rating to 1500, non FBS teams to 1200, and the scaling factor to 400. The appropriate value for K is something that I’ll want to explore empirically, but I’ll start it at 25 and go from there.

I’m using two more functions here to determine elo ratings. The first determines how Elo ratings are updated after each game, looking at each team’s pre game Elo rating and then determing their new rating baed on the outcome and the settings for \(K\) and \(V\). I also have added in the option to add a slight bump to the Elo rating for the home team in order to capture home field advantage, I’ll leave it at zero for now.

# functions for updating elo ratings\
# load function
source(here::here("functions/get_expected_score.R"))
source(here::here("functions/get_new_elos.R"))

# updating
get_new_elos
## function (home_rating, away_rating, home_margin, home_field_advantage, 
##     k, v) 
## {
##     if (home_margin > 0) {
##         home_score = 1
##     }
##     else if (home_margin < 0) {
##         home_score = 0
##     }
##     else {
##         home_score = 0.5
##     }
##     away_score = 1 - home_score
##     home_expected_score = get_expected_score(home_rating + home_field_advantage, 
##         away_rating, V = v)
##     away_expected_score = get_expected_score(away_rating, home_rating + 
##         home_field_advantage, V = v)
##     if (home_margin > 0) {
##         mov_multi = log(abs(home_margin) + 1) * (2.2/(((home_rating + 
##             home_field_advantage - away_rating) * 0.001) + 2.2))
##     }
##     else if (home_margin < 0) {
##         mov_multi = log(abs(home_margin) + 1) * (2.2/(((away_rating - 
##             home_rating + home_field_advantage) * 0.001) + 2.2))
##     }
##     else {
##         mov_multi = 2.2 * log(2)
##     }
##     home_new_rating = home_rating + (mov_multi * k * (home_score - 
##         home_expected_score))
##     away_new_rating = away_rating + (mov_multi * k * (away_score - 
##         away_expected_score))
##     return(c(home_new_rating, away_new_rating, home_expected_score, 
##         away_expected_score))
## }

The next function is what I’ll use to calculate Elo ratings based on historical data, using a set of games as an input and settings for the various options to calculate Elo ratings.

1.1 Initial Settings

Let’s run from 1970 to 2000 to see how the results change each team’s Elo rating over the course of a couple decades. This means we loop over every game, look at the result, and then update each team’s Elo rating based on the outcome and the margin of victory (as well as our other settings).

Running this gives us a pre and post game Elo rating for every team for every game over the selected time period.

1.1.1 Examples

We can see how these ratings adjust for selected teams over time. At the current settings, here is each SEC team’s Elo ratings fared over the selected time period, starting from 1985.

And here’s the Big 10.

And the Southwest Conference (RIP).

1.1.2 Results

How do we evaluate how good these Elo ratings are? First, we can look at the log loss for the predicted score for the home team (ie, the probability the home team wins) compared to the outcome for the home team.

How did we do? I’ll compare the result of the Elo ratings to a baseline model that assigns the home team a 60% chance of winning.

As we would hope, the Elo ratings outperform simply predicting that the home team will win.

We can also just look at the number of games correctly predicted, and find similar results.

1.2 Adding Mean Reversion

As it’s constructed so far, the Elo ratings for a team in one season pick up right from where they left off in the previous season. This is particularly an issue in the world college football, given that players graduate and turnover in rosters year over year is high. We shouldn’t expect, in other words, a team at the beginning of its 1980 season to be the exact same as it ended its 1979 season.

Currently, our Elo ratings do not reset in anyway, and this means we ended up being a little bit worse at the beginning of each season.

One way we can try to improve this is by regressing each team back towards the mean at the start of each season. If we think about Elo ratings in a Bayesian way, we started each team out with a mostly uninformative prior that we updated based on new information. By the end of a season we have a much better idea about how good a team is. But, between seasons, teams can change quite a bit, and we should become less certain about what we know about a team at the start of a season.

I’ll handle this by mean reverting FBS and non FBS teams slightly back to their original setting (1500 and 1200, respectively) at the start of each season. To start, I’ll set this reversion to 1/4, meaning that a team’s rating at the start of a season is 3/4 the result of its prior games and 1/4 back to the mean. Good teams will get pulled down slightly and bad teams will get pulled up.

1.2.1 Examples

How do the ratings with mean reversion compare to the ratings with no reversion?

I’ll take a team like Wisconsin and compare their pregame ratings over this time period under both version. At the start, the two ratings are the same, as Wisconsin hasn’t played enough games to shift that far from the initial setting of 1500. But over the years, we can see how reverting Wisconsin a bit at the start of each season changes its rating. Over the course of 40 odd years, this difference starts to really add up in terms of the scale.

If we look at Big Ten teams during this time period, we should see them tighten up at the beginning of each season.

1.2.2 Results

I’ll now compare the initial ratings to the updated ratings that include mean reversion. Looking at the two methods season over season, there’s not much difference between them. I could do like, a Wald test, to compare the two formally, but who has time for that.

If anything this reversion seems to be hurting the results over this time period. I wonder if we do better in the early weeks but worse in the latter?

Hmmm. Not really. But, we’ll keep reversion as a tuning parameter and see how it does with our other settings.

1.3 Adding Home Field Advantage

I’ve seen other Elo rating systems explicitly add points to teams that are at home as a means of accounting for home field advantage. Right now, I’m not doing that, so we should expect to see the home team outperform their Elo rating. Is that the case? I’ll look at the performance of the predictions from each of the models so far. Generally speaking, we don’t do as well detecting when predicting the away team is going to win.

##          Home Win
## Home Pred   no  yes
##       no  5991 3682
##       yes 2529 9750
##          Home Win
## Home Pred   no  yes
##       no  6006 3676
##       yes 2514 9756

The Elo ratings with mean reversion are basically the same overall, though both versions tend to do better when predicting that the home team will win (PPV). When the models predict that the home team will lose, they don’t do as well (NPV). Right now, we tend to under predict how much the home team wins.

One other parameter I want to toggle here is a home team advantage. I’ll start by adding a 25 point bump to the Elo score for the home team, as this roughly maps to about a 3% increase in the probability of winning. I won’t add any bonus for games played at neutral sites. I’ll now re run making this home team adjustment.

1.3.1 Results

I’ll now compare the various versions we’ve computed so far, summarizing to their performance over the entire time period we’ve looked at.

The home field advantage makes us slightly more accurate by picking up a few more home wins, but generally speaking the differences are all still pretty small.

1.4 Tuning Over All Parameters

At this point, we’ve covered a few different parameters that we can toggle when computing Elo ratings: mean reversion, home field advantage, as well as the original settings of the K factor and the scaling factor (what I have dubbed V). I’ll now tune over these directly via a grid search and see well our Elo ratings perform at various different settings.

Once that’s finished we can look at how the different ratings performed across the combination of tuning parameters.

As before, we get pretty similar results across the board. I’ll select some of the top candidates for each of these different metrics (I’m mostly looking at the log loss) and see how they do on the validation set.

1.5 Validating Selected Parameters

At the best candidates for the selected parameters, I’ll compute the Elo ratings over the time period we’ve used so far. I’ll then evaluate their performance over the next decade of games.

Of these candidate parameters, which approach did the best for the next decade of games?

Again, the results are pretty consistent, so I’ll go with K at 35, V at 400, reversion at 10%, and home field advantage at 75.

2 Elo Ratings and the Spread

All of the work so far has been focused on predicting the winner of games in terms of both the probability and the label.

But, a nice feature of the Elo rating framework is that it also extends to predicting the difference in points between the two teams. Here is the relationship between the home team’s point differential (Home Points - Away Points) and difference in pre game Elo ratings (Home Pre Elo - Away Pre Elo) from the seasons 1980 to 2000.

I’m omitting the first few years, as as the initial few seasons are being used to learn the Elo scores, but once we have a few seasons worth of data we start to see a pretty consistent relationship between the difference in Elo ratings and the game’s point differential.

2.1 Regression

What is this relationship? I’ll regress the point spread on the difference in pregame Elo ratings for each of these individual years and over all years. Then, we’ll look at the coefficent on the difference in Elo ratings.

For the model trained on most of the training set of games, the estimated effect of HOME_ELO_DIFF was about .05. The intercept is about 0.5, which indicates the expected point differential in favor of the home team which the two teams have equivalent Elo ratings. This would be higher if we weren’t accounting for home field advantage in the Elo ratings directly. The fact that we see a slight difference means that our Elo ratings suggests that we have almost, but not entirely, picked up the effect of home field advantage with our model.

The coefficient of .05 for the Elo diff means that a 20 point lead in Elo translates to about a point on the spread. This is linear, so a 200 point Elo lead translates to about a 10 point expected margin of victory.

This effect has changed a bit over time, which we can see by looking at model fit to each season. But it’s been hovering around .05 for the time period shown here.

This isn’t to say that a 200 point lead on Elo means a 10 point spread is suddenly a sure thing. We can look at the standard deviation of the residuals (sigma) from each model to get a sense of how far off, on average, predictions for the point differential were. Generally, sigma was around 16, meaning the predictions were off by 16 points on average.

That number seems laughably high, but it’s skewed somewhat by the model being really off for certain games. The in sample mean absolute error was closer to 13 points, with an Rsquared of around .38.

2.2 Simulating the Margin of Victory

What’s the point of this? In addition to using Elo ratings to predict the outcomes of games, we can also use them to simulate the expected point differential. I’ll use one game as an example illustrate.

Let’s go to a week 1 game in 2001: Wisconsin vs Virginia. After adjusting for home team advantage, Wisconsin was the clear favorite going into this game, with around a 200 point advantage in pregame Elo. As the home team, this means Wisconsin is about a 10.5 point favorite according to our model (home team advantage). That’s the point estimate, but we can simulate from the model a few thousand times and plot the distribution of the simulations to see the uncertainty around the estimate.

The median prediction was about 14 points and the 50% prediction interval for the game ranged from about 4 to 26 while the 80% interval ranged from about -6 to 36.

What was the actual score? Wisconsin 26, Virginia 17, for a point differential of 9.

So in this case, the model was pretty close, but the point is the uncertainty is generally quite high with our estimates. If we look at all games from week 1, we can see that the model got some right and some wrong - the actual margin is in red, the 80% prediction interval is in black.

That’s an 80% prediction interval. How many times did the actual score fall within that interval?

## # A tibble: 1 × 4
##   .interval in_interval     n  prop
##   <chr>     <lgl>       <dbl> <dbl>
## 1 80%       TRUE            7     1

A little over 90% of the time.

We can do this for the entire season and look to see how one simulation looks for the whole season.

3 Simulating an Entire Season

Okay, so at this point we have everything we need to try out simulating entire seasons.

To simulate a season, we simulate each game based on the pre game Elo scores for each team. Say that Team A has a 67% chance of winning based on the two teams Elo scores. In that case, I can simulate the game by pulling from a Bernoulli distribution where p =.67. Based on that result, I then compute the outcome and update the ratings. I repeat this process N times for every game, meaning that the Elo ratings are themselves updated on the basis of each simulated result. If I simulate a season 1000 times, I get a distribution of simulated outcomes for every game and team, which I can then use to obtain probabilities of specific events.

The snag with this approach is that I’m using a margin of victory multiplier to update ratings after each game. To use this approach, I need to not only simulate the winner, but also the point margin of each game. This is where the model to predict the point differential comes in.

I can simulate the outcome and margin of each game by pulling one simulation from the points model, then use this to update the ratings.

3.1 The 2001 Season

To show this works, I’ll simulate the entire 2001 season starting with the Elo ratings we computed on games from 1970-2000.

3.1.1 Simulated Elo Paths

I’ll plot the results of these simulations for every team in the Big 12. This plot shows the simulated postgame Elo ratings for each team in this conference for the 2001 season, where each line is one possible path that the team’s season might take based on the simulated results.

This ends up painting a pretty interesting picture of how each team’s season might unfold. In each simulation of a season, a team’s Elo rating is updated after each game, which affects how likely they are to win their next game, which affects their Elo rating for the next game, and so on. When we see many lines overlap, this is because there is a typical path based on a team’s schedule for how their season is likely to unfold.

Let’s zoom into the simulations for Kansas State’s 2001 season. I’ll plot Kansas State’s pregame Elo rating in every simulation across each game in this season. Going into the season, based on their ratings from the last season, they were slight favorites on the road at USC, but it could easily go the other way. The result of this game sends KState on two pretty different paths. Regardless of how they played at USC, they’re likely to win against New Mexico State (and only marginally increase their Elo rating). But, in the simulations where they beat USC, they were more likely to beat Oklahoma; when they lost to USC, they were less likely to beat Oklahoma.

After each of these games, we would be able to update our Elo rating and simulate the rest of the season. But before any games happen, summing up the simulations gives us our preseason probabilities for each matchup.

Interestingly, Kansas State had a poor year in 2001 relative to our expectations. Prior to the season, they were favorites to win 9 out of their 11 games. We’ll dive into win totals next to take a look at this.

3.1.2 Win Totals

The simulated Elo ratings can tell us the general expected story for a team’s season. But what we probably care about is just the wins. I’ll make a similar plot but show each team’s simulated win count over the course of a season.

For win totals, we probably mostly care about showing the distribution of wins for each team at the end of the season. This is the distribution of wins in every simulation.

We can put all this info into one table by counting the number of simulations in which each team fell into number of wins. That will give us our probabilities for each team’s win total going into this season.

Man, Nebraska really was given that much of a chance of running the table?

Anyway, we can look at expected wins for all teams, not just the Big 12…

3.1.3 Pre Season Results

How well did this approach do in predicting the 2001 season at the start of the season? I’ll examine this first at the game level, looking at how the simulated results did in predicting both the probability and the outcome in every matchup for this season. I’ll assess probabilities using the log loss and the outcomes using accuracy.

Overall, we were a little over 70% accurate in predicting all games from the start of the season with a log loss in the .54 range. In both cases, this is an improvement over using a simple heuristic such as the home team always wins.

We can also break this down by each week. As we would expect, in simulating the entire season, our pre season accuracy drops further into the season we go.

Next, we can look at how the model did in predicting each team’s win totals.

On average, we were off by about 2 wins - a non trivial amount - and the performance was quite a bit different by conference and team.

3.1.4 Mid Season Results

So far, we’ve looked at how well the model did in simulating each team’s season. But I’ll be planning to run this after every week of games, which will result in updated ratings, which will affect the rest of the season’s simulations.

This means teams with surprise wins/losses will have their expectations for the rest of the season changed.

Take a look at a team like Maryland that had a bunch of tossups on its schedule early into the season, with two games at the end of the year in which they were expected to lose.

But, Maryland managed to run the table through their first 8 weeks of the season, so how did their win probabilities for their later games change after week 8?

Nebraska was the heavy favorite at the beginning of the season playing at Colorado on the last week of the season, how did the odds of that game change over the course of the season as Colorado became highly rated?

Still favored to win, but by a bit less - and the model was still wrong, as Nebraska was killed in that game, but it’s less wrong than at the start.

3.1.5 Mid Season Changes

We can do this for every week within a season to see how a team’s rating and expected end of season win totals change throughout the year.

Also, looking at predictions that are updated within the season allows me to check on how well the model is doing by how far out it is predicting. As we would expect, our accuracy is highest in predicting games one week out and trailing off after that.

Updating the simulations each week allows us to check in on how a team’s expected win totals are changing after every week. Here are the simulated end of season win totals for each ACC team after weeks 3, 7, and 10.

Similarly, here’s how the Big Ten changed as Wisconsin entered with the most expected wins, only to drop 2 of their first three games. By week 11 Illinois had completed its somewhat stunning charge to the top.

And if we look at the Big 12, we can see the rise of Colorado.